Variable Selection and Dimension Reduction by Learning Gradients

High dimension data analysis has become a challenging problem in modern sciences. The diagnosis by gene expression or SNPs data arising in medical and biological sciences is a typical example and may be the most important focus of the research in the past decade. The number of variables in these data sets may be tens to hundreds of thousands. The understanding of the data structure and inference are much difficult because of the curse of the dimension. This has driven the rapid advances in the research of variable selection and dimension reduction techniques in machine learning and statistical communities that show great advantages. Variable selection is closely related to the relevance study and dates back at least to the middle of 1990’s (Tibshirani, 1996; Blum and Langley, 1997; Kohavi and John, 1997). Since then it has been rapidly developed, especially after the microarray data and text categorization draw attention of researchers. A special issue on this topic is published by Journal of Machine Learning Research in 2003. In Guyon and Elisseeff (2003) the main benefits of of variable selection were summarized to be 3-fold: improving the inference performance, providing faster and costeffective predictors, and better understanding of underlying process that generates the data. The many methods that have been proposed in the literature includes various correlation and information criteria (see Guyon and Elisseeff (2003) for a